Image processing method and apparatus

ABSTRACT

An image processing method including: obtaining points on left-eye and right-eye images to be generated from a two-dimensional (2D) image, to which a predetermined pixel of the 2D image is to be mapped, by using the sizes of holes to be generated in the left-eye and right-eye images; and generating the left-eye and right-eye images respectively having the obtained points to which the predetermined pixel of the 2D image is mapped.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims the benefit of Korean Patent Application No.10-2008-0105928, filed on Oct. 28, 2008, in the Korean IntellectualProperty Office, the disclosure of which is incorporated herein in itsentirety by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

Aspects of the present invention relate to an image processing methodand apparatus, and more particularly, to a method and an apparatus tominimize a size of a hole in a three-dimensional (3D) image generatedfrom a two-dimensional (2D) image.

2. Description of the Related Art

3D image techniques have become widely used with the development ofdigital technology. The 3D image techniques give information on depth to2D images so as to represent realistic images. 3D image techniques thatare being studied include a technique to generate a 3D image from videodata and a technique to convert a 2D image generated from video datainto a 3D image.

SUMMARY OF THE INVENTION

Aspects of the present invention provide an image processing method andapparatus to change points on left-eye and right-eye images, to which apredetermined pixel of a two-dimensional (2D) image is mapped, so as tominimize the size of a hole in a three-dimensional (3D) image generatedfrom the 2D image when the 3D image is generated from the 2D image.

According to an aspect of the present invention, there is provided animage processing method including: obtaining points on left-eye andright-eye images to be generated from a 2D image, to which apredetermined pixel of the 2D image is to be mapped, using sizes ofholes to be generated in the left-eye and right-eye images; andgenerating the left-eye and right-eye images respectively having theobtained points to which the predetermined pixel of the 2D image ismapped.

The obtaining of the points may include obtaining the points such thatan average size of one or more holes in the left-eye image and anaverage size of one or more holes in the right-eye image are equal toeach other.

The obtaining of the points may include determining the sizes of theholes using a depth value to be applied to the 2D image.

The determining of the sizes of holes may include dividing the 2D imageinto a plurality of blocks, obtaining a depth value difference betweenneighboring blocks by using depth values of the neighboring blocks, anddetermining the sizes of the hole using the depth value difference.

According to another aspect of the present invention, there is providedan image processing method including: determining an image observationpoint according to sizes of holes to be included in left-eye andright-eye images to be generated from a 2D image; and generating theleft-eye and right-eye images using the 2D image seen from the imageobservation point.

The determining of the image observation point may include determiningthe image observation point such that an average size of one or moreholes in the left-eye image and an average size of one or more holes inthe right-eye image are equal to each other.

According to yet another aspect of the present invention, there isprovided an image processing method including: extracting positioninformation on points on left-eye and right-eye images to which apredetermined pixel of a 2D image is to be mapped from meta data withrespect to video data; and mapping the predetermined pixel of the 2Dimage to the points to generate the left-eye and right-eye images byusing the position information.

The meta data includes shot information to classify frames in which acomposition of a background of a current frame is estimable using aprevious frame as a single shot, and the extracting of the positioninformation may include extracting position information to be applied toeach shot.

The extracting of the position information from the meta data mayinclude extracting position information to be applied to each frame ofthe video data.

The image processing method may further include reading the meta datafrom a disc or downloading the meta data from a server through acommunication network.

According to another aspect of the present invention, there is providedan image processing apparatus including: a position calculator to obtainpoints on left-eye and right-eye images to be generated from a 2D image,to which a predetermined pixel of the 2D image is to be mapped, usingthe sizes of holes to be generated in the left-eye and right-eye images;and a stereo rendering unit to generate the left-eye and right-eyeimages having the points to which the predetermined pixel of the 2Dimage is mapped.

According to another aspect of the present invention, there is providedan image processing apparatus including: a position calculator todetermine an image observation point according to sizes of holes to beincluded in left-eye and right-eye images to be generated from a 2Dimage; and a stereo rendering unit to generate left-eye and right-eyeimages using the 2D image seen from the image observation point.

According to still another aspect of the present invention, there isprovided an image processing apparatus including: a meta data analyzerto extract position information on points on left-eye and right-eyeimages to which a predetermined pixel of a 2D image is to be mapped frommeta data with respect to video data; and a stereo rendering unit to mapthe predetermined pixel of the 2D image to the points to generate theleft-eye and right-eye images using the position information.

According to yet another aspect of the present invention, there isprovided a computer readable recording medium to execute an imageprocessing method including: obtaining points on left-eye and right-eyeimages to be generated from a 2D image, to which a predetermined pixelof the 2D image is to be mapped, using the sizes of holes to begenerated in the left-eye and right-eye images; and generating theleft-eye and right-eye images having the points to which thepredetermined pixel of the 2D image is mapped.

According to another aspect of the present invention, there is provideda computer readable recording medium to execute an image processingmethod including: determining an image observation point according tosizes of holes to be included in left-eye and right-eye images to begenerated from a 2D image; and generating the left-eye and right-eyeimages using the 2D image seen from the image observation point.

According to another aspect of the present invention, there is provideda computer readable recording medium to execute an image processingmethod including: extracting position information on points on left-eyeand right-eye images to which a predetermined pixel of a 2D image is tobe mapped from meta data with respect to video data; and mapping thepredetermined pixel of the 2D image to the points to generate theleft-eye and right-eye images by using the position information.

According to another aspect of the present invention, there is providedan image processing method including: generating, by an image processingapparatus, left-eye and right-eye images from a two-dimensional (2D)image according to sizes of holes in the left-eye and right-eye images,wherein the holes correspond to absent portions of an object in thegenerated left-eye and right-eye images that are respectively visible onthe object when actually viewed from an image observation pointcorresponding to a left-eye and a right eye of which the left-eye andright-eye images are based, the portions being absent in the generatedleft-eye and right-eye images.

Additional aspects and/or advantages of the invention will be set forthin part in the description which follows and, in part, will be obviousfrom the description, or may be learned by practice of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

These and/or other aspects and advantages of the invention will becomeapparent and more readily appreciated from the following description ofthe embodiments, taken in conjunction with the accompanying drawings ofwhich:

FIG. 1 illustrates an operation of generating left-eye and right-eyeimages using a 2D image;

FIGS. 2A to 2C are diagrams to explain a hole generated when a point ofview in relation to a 2D image is moved;

FIGS. 3A to 3C are diagrams to explain a method of converting a 2D imageinto a 3D image in consideration of sizes of holes generated in left-eyeand right-eye images according to an embodiment of the presentinvention;

FIG. 4 is a block diagram of an image processing apparatus according toan embodiment of the present invention;

FIG. 5 is a flowchart illustrating an image processing method accordingto an embodiment of the present invention; and

FIG. 6 is a diagram to compare sizes of holes generated in left-eye andright-eye images according to an embodiment of the present invention.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Reference will now be made in detail to the present embodiments of thepresent invention, examples of which are illustrated in the accompanyingdrawings, wherein like reference numerals refer to the like elementsthroughout. The embodiments are described below in order to explain thepresent invention by referring to the figures.

FIG. 1 illustrates an operation of generating left-eye and right-eyeimages using a two-dimensional (2D) image. To obtain a stereoscopic 2Dimage, the 2D image is given depth. When a person sees an imageprojected onto a screen, the image is seen by the left and right eyes ofthe person. Here, a difference between two perceived images due to thespace between the left and right eyes is referred to as parallax. Theparallax includes a positive parallax, a zero parallax, and a negativeparallax. The positive parallax corresponds to a case where the imageappears to be formed behind the screen. In this case, the parallax isequal to or smaller than a distance between the left and right eyes.Accordingly, as the parallax increases, a stereoscopic effect thatcauses the image to seem to be located deeper than the screen isobtained.

When a 2D image appears to be formed on the plane of the screen, theparallax is zero. In this case, a viewer cannot feel the stereoscopiceffect because the image appears to be formed on the plane of thescreen. The negative parallax corresponds to a case where the imageappears to be formed in front of the screen and is generated when linesof eyes of the viewer cross each other. In this case, a stereoscopiceffect as if an object in the image protrudes from the screen isobtained.

Due to a predetermined distance in the horizontal direction between theleft eye and the right eye, the left eye and the right eye see a 2Dimage differently, which is referred to as binocular disparity. Thehuman brain combines the two different 2D images seen by the left eyeand the right eye to generate a three-dimensional (3D) image havingdepth and presence. To generate the two different 2D images seen by theleft eye and the right eye (i.e., left-eye and right-eye images), pointson the left-eye and right-eye images are known, such that apredetermined pixel of the original 2D image is mapped thereto.Furthermore, to convert the 2D image into a 3D image, the 2D image isgiven depth. The points on the left-eye and right-eye images, to whichthe predetermined pixel of the 2D image is mapped, depend on the depthto be given to the 2D image.

Referring to FIG. 1, the X-axis represents horizontal pixels of a 2Dimage frame. Points on left-eye and right-eye images to which apredetermined pixel of the 2D image frame, which corresponds to ahorizontal pixel value x on the X-axis, is mapped will now be described.

The Z axis, which is parallel to a direction in which a viewer views ascreen, represents a degree of depth given to the 2D image (i.e., adepth value). Here, the Z axis represents a virtual depth value and aScale_z axis represents a physical distance to which the virtual depthvalue is converted. A depth value corresponds to a degree of depth of animage and is used to give depth to a 2D image. In the current embodimentof the invention, the virtual depth value is a value in the range of 0to 255. Specifically, the 2D image appears to be deeper and more distantfrom the viewer as the virtual depth value decreases towards zero, andthe 2D image appears to be closer to the viewer as the virtual depthvalue increases towards 255.

A panel position corresponds to a position of the screen on which animage is formed. Accordingly, a panel position value corresponds to adepth value of the image when the parallax is zero (i.e., when the imageappears to be formed on the surface of the screen). As illustrated inFIG. 1, the panel position value may have a depth value in the range of0 to 255. When the panel position value is 255, the image included inthe 2D image frame has a depth value equal to or smaller than the depthvalue of the screen, and thus the image is appears to be formed at adistance from the viewer (i.e., formed behind the screen). This meansthat the image included in the 2D image frame has a zero parallax or apositive parallax. When the panel position value is zero, the imageincluded in the 2D image frame has a depth value equal to or greaterthan the depth value of the screen, and thus the image appears to beformed in front of the screen. This means that the image included in the2D image frame has zero parallax or a negative parallax.

In FIG. 1, the viewer takes a front view of the 2D image, and thus anX-axis value of a central point between the left and right eyes of theviewer looking at the predetermined pixel of which the horizontal pixelvalue is x is also x. To obtain the points on the left-eye and right-eyeimages, to which the predetermined pixel of which the horizontal pixelvalue is x from among the pixels of the 2D image is mapped, a distanceon the X axis between the point corresponding to the horizontal pixelvalue x and each of the points on the left-eye and right-eye images towhich the pixel corresponding to the horizontal pixel value x is mappedis referred to as Shift_x. When the distance between the left eye andthe right eye of the viewer is d, a distance between the eyes of theviewer and the screen is L, and the depth value of the predeterminedpixel of the 2D image is z, the distance between the predetermined pixelof the 2D image and the center point between the left and right eyes ofthe viewer can be represented by the sum of a difference between thedepth value of the predetermined pixel of the 2D image and the depthvalue of the panel position value and a distance between the panelposition value and the center point between the left and right eyes ofthe viewer. That is, the distance between the predetermined pixel of the2D image and the center point between the left and right eyes of theviewer is (panel position value−z)+L.

${{\left( {{{Panel}\mspace{14mu}{position}\mspace{14mu}{value}} - z} \right) + {L:\mspace{14mu}\left( {{{panel}\mspace{11mu}{position}\mspace{14mu}{value}} - z} \right)}} = {{d/2}\text{:}\mspace{14mu}{Scale\_ z}}},\begin{matrix}{{and}\mspace{14mu}{thus}\mspace{14mu}{Shift\_ x}\mspace{14mu}{corresponds}\mspace{14mu}{to}\mspace{14mu}{\frac{\left( {{{panel}\mspace{14mu}{position}\mspace{14mu}{value}} - z} \right)*\frac{d}{2}*{Scale\_ z}}{\left( {{{panel}\mspace{14mu}{position}\mspace{14mu}{value}} - z} \right) - L}.}}\end{matrix}$

When the points on the left-eye and right-eye images, to which thepredetermined pixel corresponding to the X-axis value x is mapped, arerespectively Xl and Xr, Xl and Xr can be represented according toEquation 1:Xl=x−Shift_(—) xXr=x+Shift_(—) x  [Equation 1]

As described above, according to FIG. 1, the points on the left-eye andright-eye images, to which the predetermined pixel of the 2D image ismapped, correspond to points respectively located apart from thepredetermined pixel to the left and right by a predetermined distance(Shift_x).

FIGS. 2A to 2C are diagrams to explain a hole generated when a viewpointin relation to an image is moved. An image of a predetermined object,captured by a camera or seen by a viewer, varies according to the pointat which the camera or the viewer sees the object. That is, a 2D imageof the object is generated differently according to the point at whichthe object is seen or captured. Hereinafter, the point at which thecamera or the viewer captures or sees the predetermined object isreferred to as an image observation point.

FIGS. 2A, 2B and 2C illustrate images of the same building, seen by theviewer from different image observation points. FIG. 2B illustrates a 2Dimage of the building, captured by the camera or seen by the viewer whenthe camera or the viewer captures or sees the building from a secondimage observation point in an upper part of FIG. 2B, and illustrates across-sectional view of the building in a lower part of FIG. 2B.

FIG. 2A illustrates a 2D image of the building, captured by the cameraor seen by the viewer when the camera or the viewer sees the buildingfrom a first image observation point. The 2D image illustrated in FIG.2A includes an image of a predetermined part of the building, which isnot shown in the 2D image illustrated in FIG. 2B. That is, the 2D imageillustrated in FIG. 2A includes a part shown in a dotted line thatindicates an image of a left part of the building, which is seen fromthe point of view of the viewer at the first image observation point.

FIG. 2C illustrates a 2D image of the building, captured by the cameraor seen by the viewer when the camera or the viewer sees the buildingfrom a third image observation point. The 2D image illustrated in FIG.2C includes a part shown in a dotted line that indicates an image of aright part of the building, which is seen from the point of view of theviewer and is not shown in the 2D image illustrated in FIG. 2B. That is,the image seen by the viewer varies according to the image observationpoint.

As described above, pixels of the 2D image are mapped to predeterminedpoints of the left-eye and right-eye images according to depth values ofthe pixels in order to generate a 3D image from the 2D image. When theleft-eye and right-eye images are generated using the 2D imageillustrated in FIG. 2B, the left and right parts of the building, whichare respectively included in the images of FIG. 2A or 2C and are to beincluded in the 3D image, is not seen, because the left-eye andright-eye images are generated using only the 2D image illustrated inFIG. 2B. Accordingly, parts of the left-eye and right-eye imagesgenerated from the 2D image illustrated in FIG. 2B, which correspond tothe left and right part of the building, have holes.

FIGS. 3A to 3C are diagrams to explain a method of converting a 2D imageinto a 3D image in consideration of sizes of holes generated in left-eyeand right-eye images, according to an embodiment of the presentinvention. FIG. 3B illustrates that points on the left-eye and right-eyeimages, to which a predetermined pixel of the 2D image is mapped,correspond to points located apart from the predetermined pixel of the2D image to the left and right by a predetermined distance (i.e., pointslocated apart from the predetermined pixel by Shift_x in Equation 1), asdescribed above with reference to FIG. 1. However, it is understood thataspects of the present invention are not limited thereto. For example,according to some aspects, the 2D image can be used as the left-eye orright-eye image.

FIG. 3C illustrates a case where the left-eye image to be generated fromthe 2D image is identical to the 2D image. The 2D image is used as theleft-eye image and, thus, only the right-eye image is newly generated.In this case, the point on the right-eye image, to which thepredetermined pixel of the 2D image is mapped, corresponds to a pointlocated apart from the predetermined pixel of the 2D image to the rightby 2*Shift_x.

Similarly, the 2D image may be used as the right-eye image and only theleft-eye image may be newly generated. FIG. 3A illustrates such a casewhere the 2D image is used as the right-eye image. In this case, thepoint on the left-eye image, to which the predetermined pixel of the 2Dimage is mapped, corresponds to a point located apart from thepredetermined pixel of the 2D image to the left by 2*Shift_x.

Aspects of the present invention consider a method of generatingleft-eye and right-eye images using a 2D image such that a predeterminedpixel of the 2D image is mapped to points located apart from thepredetermined pixel to the left and right by distances different fromeach other as well as a method of generating the left-eye and right-eyeimages such that the predetermined pixel of the 2D image is mapped topoints located apart from the predetermined pixel to the left and rightby the same distance (i.e., equal to Shift_x).

When the points on the left-eye and right-eye images, to which thepredetermined pixel corresponding to the X-axis value x is mapped, arerespectively Xl′ and Xr′, Xl′ and Xr′ can be represented according toEquation 2:Xl′=x−(1−a)*Shift_(—) x,Xr′=x+(1+a)*Shift_(—) x,  [Equation 2]where a is a rational number greater than or equal to −1 and less thanor equal to 1.

When a is zero in FIG. 3, Equation 2 becomes identical to Equation 1. Inthis case, the left-eye and right-eye images are generated from the 2Dimage such that the predetermined pixel of the 2D image is mapped atpoints of the left-eye and right-eye images located apart from thepredetermined pixel to the left and right by the same distance (i.e.,equal to Shift_x), as illustrated in FIG. 3B. However, when a is 1,Xl′=x and Xr′=x+2*Shift_x are obtained using Equation 2. Accordingly,the left-eye image generated from the 2D image becomes identical to the2D image, as illustrated in FIG. 3C. In this case, the right-eye imagecorresponds to an image to which the predetermined pixel of the 2D imageis mapped at a point located apart from the predetermined pixel to theright by a distance twice the distance in the case of FIG. 3B (i.e.,2*Shift_x). Similarly, when a is −1, Xl′=x−2*Shift_X and Xr′=x areobtained using Equation 2. That is, the right-eye image is identical tothe 2D image and the left-eye image corresponds to an image to which thepredetermined pixel of the 2D image is mapped at a point located apartfrom the predetermined pixel to the left by a distance twice thedistance in the case of FIG. 3B (i.e., 2*Shift_x), as illustrated inFIG. 3A.

A method of acquiring the value a will now be explained. It is assumedthat a single frame of the 2D image is divided into W*H blocks includingW horizontal blocks and H vertical blocks. A block can be a single pixelor a set of a plurality of pixels. When a depth value of a horizontallyith and vertically jth block is Z(i, j), depth values of all the blockscorrespond to Z(1, 1) through Z(W, H). The sizes of holes in theleft-eye and right-eye images generated from the 2D image can bedetermined using a depth value difference between blocks. When a depthvalue difference between a current block and the next block to the rightof the current block is G, G can be represented according to Equation 3:G(i,j)=Z(i+1,j)−Z(i,j),  [Equation 3]where Z(i+1, j) is a depth value of a horizontally (i+1)th andvertically jth block and Z(i, j) is a depth value of the horizontallyith and vertically jth block.

Functions Hl and Hr with respect to hole sizes can be obtained fromEquation 4 below, which uses Equation 3:Hl=ΣG(i,j)^2,G(i,j)>0  [Equation 4]Hr=ΣG(i,j)^2,G(i,j)<0,where Hl and Hr respectively represent functions with respect to thesizes of holes in the left-eye and right-eye images. The relationshipbetween the functions Hl and Hr and a can be represented according toEquation 5:a=1−2[Hl/(Hl+Hr)].  [Equation 5]

As described above, according to the current embodiment of theinvention, the left-eye and right-eye images can be generated using the2D image in consideration of the depth value of the 2D image such thatthe predetermined pixel of the 2D image is mapped to the points locatedapart from the predetermined pixel to the left and right by distancesthat may vary from each other.

FIG. 4 is a block diagram of an image processing apparatus 400 accordingto an embodiment of the present invention. Referring to FIG. 4, theimage processing apparatus 400 includes a video data decoder 410, a metadata analyzer 420, a mask buffer 430, a depth map generator 440, astereo rendering unit 450, a position calculator 470, and an output unit460 to display a 3D image generated in a 3D format on a screen. However,it is understood that the output unit 460 is not necessarily included inthe image processing apparatus 400 in all embodiments of the presentinvention. The image processing apparatus 400 may be a television, acomputer, a mobile device, a set-top box, a gaming system, etc. Theoutput unit 460 may be a cathode ray tube display device, a liquidcrystal display device, a plasma display device, an organic lightemitting diode display device, etc. Moreover, while not required, eachof the units 410, 420, 430, 440, 450, 460, 470 can be one or moreprocessors or processing elements on one or more chips or integratedcircuits.

To convert a 2D image into a 3D image, a method of obtaining a depthvalue of a current frame by using a motion difference between a previousframe and the current frame and/or a method of extracting depthinformation to be applied to the 2D image from meta data and using thedepth information may be used. The image processing apparatus 400illustrated in FIG. 4 employs the latter method. However, it isunderstood that aspects of the present invention are not limited theretoand can be applied, for example, to the method of obtaining the depthvalue of the current frame using a motion vector.

In the current embodiment of the present invention, meta data includesinformation to convert video data frames of a 2D image into a 3D image.Video data includes a series of frames and, thus, meta data to generatea depth map includes information on the frames. The information on theframes may include information to classify the frames according to apredetermined standard. For example, when a bundle of similar frames isreferred to as a unit, the frames of the video data can be classifiedinto a plurality of units (for example, a plurality of shots). In thecurrent embodiment of the present invention, the meta data to generatethe depth map includes information to classify the frames of the videodata into predetermined units.

In particular, when the composition of a current frame can be estimatedusing a previous frame since the current and previous frames havesimilar compositions, the frames having similar compositions arereferred to as a shot. The meta data includes shot information toclassify the frames of the video data into shots. When the compositionof a current frame is different from that of a previous frame sincecompositions of frames are remarkably changed, the current frame and theprevious frame are classified into different shots. In this case, aspecific value a or specific values Xl′ and Xr′ can be applied to allthe frames included in each shot or different values a or differentvalues Xl′ and Xr′ can be applied to respective frames.

When a disc (not shown) on which video data of a 2D image and meta datawith respect to the video data are recorded in a multiplexed manner or aseparated manner is loaded into the image processing apparatus 400, thevideo data decoder 410 and the meta data analyzer 420 respectively readthe video data and the meta data from the disc. The meta data can berecorded in at least one of a lead-in zone, a user data zone and alead-out zone of the disc. While not required, the image processingapparatus 400 can include a drive to read the disc directly, or can beconnected to a separate drive. However, it is understood that aspects ofthe present invention are not limited thereto. For example, in someaspects, the image processing apparatus 400 may further include acommunication unit (not shown) that communicates with an external serveror an external terminal through a communication network. Accordingly,the image processing apparatus 400 can download video data and/or metadata with respect to the video data from the external server or theexternal terminal through the communication unit and store the videodata and/or the meta data in a local storage unit (not shown) of theimage processing apparatus 400. The local storage unit may be a volatilememory (such as RAM) or a non-volatile memory (such as ROM, flashmemory, or a hard disk drive) Furthermore, according to other aspects,the image processing apparatus may receive the video data and/or themeta data from any type of external storage media (such as a flashmemory, a universal serial bus memory, etc.).

The video data decoder 410 reads the video data from the externalstorage medium (such as the disc) or the local storage unit and decodesthe read video data. The meta data analyzer 420 reads the meta data withrespect to the video data from the disc or the local storage unit andanalyzes the read meta data. When the video data is recorded on a disc,the meta data analyzer 420 may extract a disc identifier to identify thedisc on which the video data is recorded from the meta data and a titleidentifier to indicate which title includes the video data among aplurality of titles recorded in the disc and determines which video datais related to the meta data.

Furthermore, the meta data analyzer 420 parses depth information to beapplied to the current frame from the meta data and sends the depthinformation to the depth map generator 440. The depth information may beclassified into background depth information and object depthinformation. The background depth information represents a depth valueof a background included in a frame and the object depth informationrepresents a depth value of an object included in the frame. The objectis an individual object other than the background in the frame.Information on a mask may be defined as object region information for anobject included in a currently displayed frame. In this case, the maskbuffer 430 temporarily stores the mask to be applied to the frame. Themask can have a portion corresponding to the object that is in a colordifferent from that of another portion of the mask or is perforatedalong the shape of the object.

The depth map generator 440 generates a depth map of the frame using thebackground depth information and the object depth information receivedfrom the meta data analyzer 420 and/or the mask received from the maskbuffer 430. The depth map generator 440 respectively generates a depthmap for the background and a depth map for the object using the metadata and sums the depth map for the background and the depth map for theobject to generate the depth map for the single frame. Moreover, thedepth map generator 440 sends the generated depth map to the stereorendering unit 450.

The position calculator 470 obtains points on left-eye and right-eyeimages to which a predetermined pixel of the 2D image is to be mapped.For example, the position calculator 470 obtains Hl and Hr usingEquations 3 and 4 when a is zero (i.e., when the points on the left-eyeand right-eye images to which the predetermined pixel of the 2D image ismapped are located apart from the predetermined pixel of the 2D image byShift-x, as illustrated in FIG. 3B). The position calculator 470 appliesHl and Hr to Equation 5 to obtain the value a and applies the value a toEquation 2 to obtain Xl′ and Xr′. The position calculator 470 sends Xl′and Xr′ to the stereo rendering unit 450.

Although the position calculator 470 obtains Xl′ and Xr′ in the currentembodiment of the present invention, it is understood that aspects ofthe present invention are not limited thereto. For example, in otheraspects, the meta data with respect to the video data may includeinformation on Xl′ and Xr′. In this case, the meta data analyzer 420 canextract the points on the left-eye and right-eye images to which thepredetermined pixel of the 2D image is to be mapped (i.e., Xl′ and Xr′)from the meta data and sends Xl′ and Xr′ to the stereo rendering unit450.

The stereo rendering unit 450 generates the left-eye and right-eyeimages using the video image received from the video data decoder 410and the depth map received from the depth map generator 440.Accordingly, the stereo rendering unit 450 generates a 3D image in a 3Dformat including both the left-eye and right-eye images. Specifically,the stereo rendering unit 450 receives Xl′ and Xr′ that represent thepoints on the left-eye and right-eye images to which the predeterminedpixel of the 2D image is to be mapped from the position calculator 470or the meta data analyzer 420, generates the left-eye image such thatthe predetermined pixel of the 2D image is mapped to the pointcorresponding to Xl′, and generates the right-eye image such that thepredetermined pixel of the 2D image is mapped to the point correspondingto Xr′. The stereo rendering unit 450 generates a 3D image in a 3Dformat using the generated left-eye and right-eye images. The 3D formatmay include a top-down format, a side-by-side format, and/or aninterlaced format. The stereo rendering unit 450 sends the 3D image inthe 3D format to the output unit 460.

The output unit 460 sequentially displays the left-eye and right-eyeimages on the screen. A viewer recognizes that images are continuouslyand sequentially reproduced when the images are displayed at a framerate of at least 60 Hz on the basis of one eye of the viewer. Thus, adisplay device displays the images at a frame rate of at least 120 Hz inorder to combine the images seen by the left and right eyes of theviewer such that the viewer recognizes the combined images as a 3Dimage. Accordingly, the output unit 460 sequentially displays left andright images included in a frame at least every 1/120 of a second.

FIG. 5 is a flowchart illustrating an image processing method accordingto an embodiment of the present invention. Referring to FIG. 5, theimage processing apparatus 400 illustrated in FIG. 4 determines whethermeta data includes information on points on left-eye and right-eyeimages to which a predetermined pixel of a 2D image is to be mapped inoperation 510. Accordingly, the image processing apparatus 400illustrated in FIG. 4 extracts a value a from the meta data when themeta data includes the value a.

When the meta data does not include the value a (operation 510), theimage processing apparatus 400 calculates the value a in operations 520and 530. To achieve this, the image processing apparatus 400 divides asingle frame into a plurality of blocks and obtains an inter-block depthdifference using a depth value difference between a current block andthe next block adjacent (for example, to the right of) the currentblock. The image processing apparatus 400 obtains functions Hl and Hrwith respect to hole sizes using the inter-block depth difference inoperation 520. Accordingly, the image processing apparatus 400 obtainsthe value a by using Hl and Hr in operation 530 and obtains the pointsXl′ and Xr′ on the left-eye and right-eye images to which thepredetermined pixel of the 2D image is to be mapped using the value a inoperation 540.

FIG. 6 is a diagram to compare sizes of holes generated in left-eye andright-eye images according to an embodiment of the present invention.Upper images illustrated in FIG. 6 respectively represent hole sizesincluded in left-eye and right-eye images generated when a is zero. Thesize of a hole is represented by the thickness of a pixel including thehole in FIG. 6. It can be seen from FIG. 6 that the size of a holegenerated in the left-eye image is greater than the size of a holeincluded in the right-eye image when a is zero. When the sizes of theholes generated in the left-eye and right-eye images are different fromeach other, a 3D image obtained when the left-eye image and theright-eye image are alternately displayed may seem unnatural to aviewer.

Lower images illustrated in FIG. 6 respectively represent hole sizesincluded in left-eye and right-eye images when a is 0.23. It can be seenfrom FIG. 6 that the size of the hole generated in the left-eye imagewhen a is 0.23 is smaller than the size of the hole generated in theleft-eye image when a is zero, and the size of the hole generated in theright-eye image when a is 0.23 is greater than the size of the holegenerated in the right-eye image when a is zero. As illustrated, when ais 0.23, the sizes of the holes generated in the left-eye and right-eyeimages are equal to each other. In this case, the 3D image obtained whenthe left-eye and right-eye images are alternately displayed seemsnatural to the viewer.

While not restricted thereto, aspects of the present invention can alsobe embodied as computer-readable code on a computer-readable recordingmedium. The computer-readable recording medium is any data storagedevice that can store data that can be thereafter read by a computersystem. Examples of the computer-readable recording medium includeread-only memory (ROM), random-access memory (RAM), CD-ROMs, magnetictapes, floppy disks, and optical data storage devices. Thecomputer-readable recording medium can also be distributed overnetwork-coupled computer systems so that the computer-readable code isstored and executed in a distributed fashion. Aspects of the presentinvention may also be realized as a data signal embodied in a carrierwave and comprising a program readable by a computer and transmittableover the Internet. Moreover, while not required in all aspects, one ormore units of the image processing apparatus 400 can include a processoror microprocessor executing a computer program stored in acomputer-readable medium.

Although a few embodiments of the present invention have been shown anddescribed, it would be appreciated by those skilled in the art thatchanges may be made in this embodiment without departing from theprinciples and spirit of the invention, the scope of which is defined inthe claims and their equivalents.

What is claimed is:
 1. An image processing method comprising: obtaining,by an image processing apparatus, points on left-eye and right-eyeimages to be generated from a two-dimensional (2D) image, to which apredetermined pixel of the 2D image is to be mapped, using sizes ofholes to be generated in the left-eye and right-eye images; andgenerating, by the image processing apparatus, the left-eye andright-eye images respectively having the obtained points to which thepredetermined pixel of the 2D image is mapped.
 2. The image processingmethod as claimed in claim 1, wherein the obtaining of the pointscomprises obtaining the points such that an average size of one or moreholes in the generated left-eye image and an average size of one or moreholes in the generated right-eye image are equal to each other.
 3. Theimage processing method as claimed in claim 1, wherein the obtaining ofthe points comprises determining the sizes of the holes using a depthvalue to be applied to the 2D image.
 4. The image processing method asclaimed in claim 3, wherein the determining of the sizes of the holescomprises: dividing the 2D image into a plurality of blocks; obtaining adepth value difference between neighboring blocks by using depth valuesof the neighboring blocks; and determining the sizes of the holes usingthe depth value difference.
 5. The image processing method as claimed inclaim 1, wherein the obtaining of the points comprises: obtaining thepoint on the left-eye image according to Xl′=x−(1−a)*Shift_x; andobtaining the point on the right-eye image according toXr′=x+(1+a)*Shift_x, where X′ is a location value of the point on theleft-eye image, Xr′ is a location value of the point on the right-eyeimage, x is a location value of the predetermined pixel on the 2D image,Shift_x is a value corresponding to a shift of the predetermined pixelfrom x in the left-eye and right-eye images, and a is a value in a rangefrom −1 to 1 corresponding to the sizes of the holes to be generated. 6.The image processing method as claimed in claim 5, wherein a is obtainedfrom meta data corresponding to the 2D image.
 7. The image processingmethod as claimed in claim 1, wherein the holes correspond to absentportions of an object and/or a background in the generated left-eye andright-eye images that are respectively visible on the object whenactually viewed from image observation points respectively correspondingto a left-eye and a right eye of which the left-eye and right-eye imagesare based, the portions being absent in the generated left-eye andright-eye images.
 8. An image processing apparatus comprising: aposition calculator to obtain points on left-eye and right-eye images tobe generated from a two-dimensional (2D) image, to which a predeterminedpixel of the 2D image is to be mapped, using sizes of holes to begenerated in the left-eye and right-eye images; and a stereo renderingunit to generate the left-eye and right-eye images respectively havingthe obtained points to which the predetermined pixel of the 2D image ismapped.
 9. The image processing apparatus as claimed in claim 8, whereinthe position calculator determines the sizes of the holes using a depthvalue to be applied to the 2D image.
 10. The image processing apparatusas claimed in claim 9, wherein the position calculator divides the 2Dimage into a plurality of blocks, obtains a depth value differencebetween neighboring blocks by using depth values of the neighboringblocks, and determines the sizes of the holes by using the depth valuedifference.
 11. The image processing apparatus as claimed in claim 8,wherein the position calculator obtains the point on the left-eye imageaccording to Xl′=x−(1−a)*Shift_x, and obtains the point on the right-eyeimage according to Xr′=x+(1+a)*Shift_x, wherein Xl′ is a location valueof the point on the left-eye image, Xr′ is a location value of the pointon the right-eye image, x is a location value of the predetermined pixelon the 2D image, Shift_x is a value corresponding to a shift of thepredetermined pixel from x in the left-eye and right-eye images, and ais a value in a range from −1 to 1 corresponding to the sizes of theholes to be generated.
 12. A non-transitory computer readable recordingmedium encoded with computer instructions for executing the method ofclaim 1 and implemented by the image processing apparatus.